Fix Qwen2Tokenizer ignoring pretokenize_regex (Qwen3.6 / Qwen3.8 wrong token IDs) - #49077

Fixes #49066

What was wrong

Qwen2Tokenizer.__init__ always built the Split pre-tokenizer from the module-level PRETOKENIZE_REGEX, even when tokenizer_config.json carried a different pretokenize_regex.

That was invisible for Qwen2 / Qwen3 because the constant matched the published file. Qwen3.6 and Qwen3.8 refine the pattern with \p{M} (combining marks). Loading those checkpoints with AutoTokenizer dropped the marks into the punctuation branch, so Hindi / Thai / Arabic text got 50–75% more tokens than the published tokenizer.json, and save_pretrained baked the wrong pattern into derived repos.

I reproduced the issue's Hindi sample on current main: published IDs [58069, 84237, 150104, 153348, 184642, 235886], loaded IDs differed. After this change they match.

What I changed

• Read pretokenize_regex from kwargs (comes from tokenizer_config.json via from_pretrained) and use that for the Split pattern when present.
• Fall back to PRETOKENIZE_REGEX when the key is missing, so older Qwen2 / Qwen3 checkpoints without it keep the same tokenization.
• Added a small regression test that checks the kwarg path vs the default constant.

I deliberately did not update the module constant itself — that would change default tokenization for older checkpoints that omit the config key.

How I checked

• New unit test: test_honors_pretokenize_regex_from_kwargs — passed.
• Hub repro from Qwen2Tokenizer overwrites pre_tokenizer from tokenizer.json. Wrong token IDs for all Qwen3.6 / Qwen3.8 models (v4→v5 regression) #49066 against Qwen/Qwen3.8-Flash-Next (Hindi / Thai / Arabic / Latin) — IDs match published tokenizer.json.
• save_pretrained round-trip keeps \p{M} in both tokenizer_config.json and tokenizer.json.
• Spot-checked Qwen/Qwen2.5-0.5B-Instruct (no pretokenize_regex in config) still loads.

Happy to adjust if you'd rather pull the pre-tokenizer straight off tokenizer.json in TokenizersBackend instead — this scoped fix matched what the issue suggested for Qwen2Tokenizer.